Calibration of Expert Judgments in Counterterrorism Risk Assessment

The goal of this research is to develop and verify elicitation methods to assess counterterrorism values that could be of interest for the purpose of planning defensive strategies (but are difficult to estimate because of insufficient historical data) by combining the judgments of multiple domain experts. In particular, the project focused on producing reliable estimates from the opinions of multiple experts by reducing bias and overconfidence. To achieve this, we generated a consensus probability distribution for any given quantity of interest by weighting the experts’ judgments based on their performance. First, we created a list of seed variables for use in this process, whose values are verifiable using existing terrorism databases, so that the true values of the seed variables could be used to assess the experts’ performance. The experts were then asked to provide their opinions in the form of a 90% prediction interval for each quantity of interest (including the median, 5 th percentile, and 95 th percentile).